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1. Invention Name 

A Method and an Apparatus for Synchronous Mirrored Data Protection at Any Distance 

2. Inventors 

AlracWinokur 

3. Background 

One of ihe more effective methods to protect data is by maintaining a mirror miage of 
aie data at a secondary location. For the purpose of this invention we shall 
that the data to be protected is maintained on Direct Access Storage Devices pASD) 
1 e disks both in the primary and the secondary sites, however without loss of 
generality the data can be maintained in computer memory, magnetic tapes, or any 
odier device diat can storage data. 

■ Thus a typical data mirroring is achieved by taking evwy write operation to Gie 
orimary DASD where the data is originally to be stored and rephcatmg it to the 
secondary DASD where the mirrored data is maintained. A typical non-mirroied 
write ooeration consists ofthe following steps: , - \ 

1. iEtt application running on an appUcation server (a server based application) - 

issues a write command to the DASD. . 

2 Once the DASD completes executing the write command it issues an 

■ acknowledge to the server based appUcation notifying it of a successful 
execution of the write command , . , iv_ ^..^ 

3 The write command is considered successfully completed only after the server 
based application receives the acknowledge from the DASD 

When the written data is to be protected using mirroring, a write operation is 
augmented by additional steps to implement the minonng protection dependmg on 
the mirrorii^ strategy. 

For synchronous mirroring tiie steps are as following: ,„„i™rir.n 
1 A server based application issues a write command to flie mirronng apphcabon. 
2". The mirroring application issues this a write command to fliepnmaryDAiU 

and to the secondary DASD. 
• 3 Once bottiDASDs complete executing tiiewnte command th^ issue an 

acknowledge to ti»e minoring application notifying it of a successfiil execution 

of the write conunand 
4. Only .after receiving both acknowledgments the minoring application issues an 

acknowledge to tiie server based application »„«,^eeT«Pr 
5 The vwite ootamsad is considered successfiiUy completed only after the server 

based appUcation receives flie admowledgement from flie mirronng 

application. 

For asynchronous mirroring tiie steps look as foUowing: , 

1 A server based application issues a write command to flie mirrormg awJwation. 
2. The miiTOiing application issues a write command to flie prmaary DAitU 
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3 The mirroring application stores the write command in its internal memory. 

4 Once the primary DASDs completes executing the write command it issues an 
acknowledge to the mirroring application notifying it of a successful execution 
of the write command 

5 When the miiroring application receives the acknowledgments from the 
primary DASD the mirroring application issues an acknowledge to the server 
based application 

6 The write command is considered successfully completed only after the server 
based application receives the acknowledge from the mirroring application 

Asynchronously with steps 2 to 5 and at some point of time following step 3 above 
the mirroring application does the following rk a cri 

2. 1. The mirroring application issues the write command to the secondary UA£>u 
based on the data stored in its internal memory (see step 3 above) 

2.2. The mirroring application receives an acknowledgment from the secondary 
DASD , 

2.3. The mirroring application deletes from its internal memory the write command 

Note 1: The mirroring application can run-on the primary DASD, the application 
server, or on a computing appUance which is located between the application server 
and die primary DASD. 

Note 2; Typically the mirroring application is transparent to the server based 
application, I.e, die server ba$ed application is not aware tiiat it issues its vmte 
operations to a nurroring application ratiier than to the primary DASD itself. 

Unlike a^nchronous mirroring, tiie synchronous mirroring guaranties tiiat if tiie write 
operation completed successfully at tiie server based application, flie written data is 
secured at both primary and secondary DASD location. Asynchronous mirrormg 
.guaranties a successfiil completion at tiie primary DASD only. On the other hand the 
latency of a write operation in tiie asynchronous scenario is smaUer tiiaui for the 
synchronous niirroring because flie mirroring application does not need to delay 
sending an acknowledgment to ttie application tiU it receives tiie acknowledgement 
from botii tiie primary and tiie remote DASD. This delay is proportional to tiie 
distance between flie primary location and tiie secondary (mirrored) location and rt is 
of a considerable magnitude if tiie secondary DASD resides at a remote distance from 
the primary DASD. 

A disaster scenario is a situation where ttie entire data center loses it ability to 
fimction and all tiie data it maintains is (at least temporarily) lost. Disasters occur as a 
result terrorist attacks, earthquakes, floods, major power outages ate. To protect data 
m case of a disaster, a replica of tiie data has to be maintamed at a remote location (ot 
typically more tiien 200 miles) using mirroring technology. Neitiier of tiie above 
described mirroring strategies provides adequate data protection solution m case of a 
disaster If synchronous mirroring is to be used tiian tiie secondary site has to be m 
close vicinity to tiie primary site. If asynchronous mirroring is to be used tiien m case 
of flie disaster flie data associated witii flie last write operations will be lost depending 
on when flie mirroring application issues a write operation to tiie remote location 
(steps 2.1.to step 2.3. above) i.e. all data for which an acknowledge was received from 
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a primsiry DASD, but was not received yet fiom the remote DASD is assumed to be 
lost 

In this invention we provide a solution which enables to protect data in case of data 
disaster at any distance without loosing the data of the last write operations pnor to 
the disaster event 

4. Problem Definition 

A synchronous mirroring schema guaranties that in case of a disaster no data is lost 
However to obtain good performance synchronous mirroring can be used over short 
distances (local nurroring) between die primary and the secondary sites. 
Asynchronous mirroring on flie other hand, can not guarantee no lose of data 
associated widi the last write operations prior to a disaster. However usmg 
asynchronous mirroring one can maintain a mirror site at any distance (remote 
minor). 

In this invention we propose an apparatus and a mefliod to enable a full and complete 
data protection at a secondary site in case of a disaster event at the primary site, 
regardless of the distance between the two sites, with performance charactenstics of a 
local niirror. 

5. The proposed solution 

The proposed solution consists of four components: 

1. The protection machine. A general purpose computing machme like Lmux server 

for example. 

2. A specialized "black box" apparatus. 

3. A recovoy machine 

4. A set of mirroring algorithms. 



5.1 The Protection Machine 

The protection machine is any general purpose server fliat can run any program. It 
consists of a CPU, memory, internal busses, storage, communication ports, and m 
particular commiuiication ports of the typ6 over which appUcation servers 
communicate tiieir I/O traffic wifli tiie DASD. These ports wiU typically be Fibre 
channel ports or IP ports. In addition tiie protection machine contams a number of 
USB or USB like standard ports. These ports are categorized by providmg high m& 
bandwidth to the devices tiiey connect to and by providing also electnc power to Uiese 
devices. 

In diis hnplementation the protection machine will connect in band on die 1/0 _ 
communication lines through die VO communication ports between die aPPtication 
server and tiie mirroring application assumed to run on die primary (local) DASD as 
described in figure 1. In oflief unplementations die protection machine can be 
connected between the machine running the mirroring ^phcation, close to fliat 
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machine, and the remote DASD. In addition the protection machine will connect 
through its USB ports to the black box apparatus. 

5.2 Black Box Apparatus 

The black box apparatus consists ofthe following components 

1 Flash memory or any other persistent memoiy similar to a disk on key device 

Application p^,^ Se^n^r, 

server Protection DASD i^^^^^ 

machine 




I/O comm. line 



Black box 
apparatus 



Figure 1 



2. A rechargeable battery powered by the USB port. 

3. A homing device powered by the rechargeable battery. 

4! A reinforced enclosure, similar.to the technology used in airplanes. 

The black box apparatus is designed to survive any catastrophic disaster like 
explosion, &e and floods, protecting all the data stored in the flash memory similar to 
an airplane's black box. When it experiences a power outage, the hommg device 
starts transmitting a radio signal to ease the location ofthe apparatus (m case an 
e:q)losion causmg the demolition ofthe site for example). In addition the design of 
the box is such diat the persistent memory modules within the box can be easUy 
removed and placed in another black box similar to the method they can be removed 
from a digital camera's for example. This feature is needed for the case where the 
USiB port itself is damaged in a disaster. 

5.3 The Recovery Machine 

The recovery machine is any general purpbse server similar to the protection machine 
and of similar configuration. 

In this implementation it is located at Uie remote location connects to tiie remote 
DA5D. 

5.4 The Algorithms 

The algoriflims m based on the assumption that all write operations issued by tiie 
application server to flie primary DASD pass through or are at least visible by the 
protection machine. The algorittuns consists ofthe following two separate phases 

1. The data protect algorithm 

2. The disaster recovery algorithm 
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Algorithms Overview 

A fuU and complete data protection to a secondaiy site located at any distance js 
achieved by executing asynchronous mirror to the remote site of all date to be 
orotected To guarantee that no data is lost in die event of a disaster, all data which 
was not yet secured to the remote site is maintained locally in the black box apparatos. 
Once this data is written by the miiroring application to die remote location it can be 
discaided from the black box apparatus. The black apparatus and its memoiy is 
design to withstand any disaster thus protecting the data it hosts. Smce data is kept m 
the black box device only until it is written to the remote site the required memoiy 
needed to host this data within the black box is of a limited size. 

The memoiy of the black box apparatus is managed by protection machine Every 
write operation from the application server to the mirroring ?PP*'*^^*'5S.^'fonLin« 
forwarded to the protection machine. This is accomphshed m one of the followmg 

three methods: 

1 The protection machine is connected between die application server and die 
■ minwing application as in figure 1, flius all write operations pass through it 
2. The protection machine tabs on the line between the apphcation server and the 

mirroring application 
3 The protection machine connects to the mirronng application and the mirrormg 
application forward every write operation to the protection machine. 

Once accepted by die protection machine, die protection machine store the write 
operation, its associated data, and some additional information (see next section) to 
the blade box memoiy. 

The protection machine is also responsible for freeing memoiy space in the black box 
device in one ofdie three following possible methods: 

1 Whenever a new data is to be stored and the memory IS full, the protection 

■ machine wUl free memory by discarding from it the data associated wifli least 
resent write operation. In diis implementation we assume that the black box 
memoiy is large enough to guarantee that eveiy write operation is always wntten 
to the remote DASD before it is discarded from die black box memory. 

2 Whenever flie mirroring apphcation receives a positive acknowledgment from the 
remote DASD signaling the data associated with some wite operation was 
successfully stored tiiere, die mirroring application also forward this 
acknowledgement to the protection machine. Based on tins acknowledgment the 

■ Protection machine will identify which write operation was succe^Uy 
completed on bofli the primary and remote DASD, and will discard its 
corresponding data from die black box memoiy. 

In another implementation the black box apparatiis connects directly to die same 
machine running die mirroring application. In tiiis case no protection machine is 
required. TTie miiroring application will use die black box apparabB memoiy direcUy 
as its internal memoiy where it stores die write operations m step 3 ol tbe 
asynchronous mirror algorithm on page 1. 

When disaster occurs, die black box apparatiis is located, if necessmy, wifli fee aid of 
a homing device hosed on die radio signal it transmits. It can flien be shipped to a 
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remote location and connected to tbe recovery machine. The recovery machme wHl 
read the data associated with each write operation from the black box memory in the 
order in which the write data was initially stored and update the remote DASD. 

Note; There is no need to keep track which updates where already written to the 
"^^te DASD by tire mirroring application since applying tiie same update more than 
once does not impact tiie consistency of tiiedata.^ 

Note; If the USB connector itself is damaged it is possible to dismanUed the box 
extract its flash mranaory and install it in another box. 

In another implementation the black box apparatus can connect to any machine which 
is connected by communication line to the recoveiy machine and transmit the content 
of iS^Sry encrypted, using tbe communication lines through Internet for exarnple. 
The unique program needed to transmit the data to the remote location will iteelf be 
stored in die black box memory, so for this particular ^^^f^^^^'f^^^ 
having Internet access and USB port no additional requirements need to be imposed 
on tiiat^general purpose machine. 

Data Structures . i. j 

The major data structure maintained by tiie algorithms presented here is 0* 
associated with each write operation which is stored witim the memory of ^a^. 
totappanitois. For each write operation the following data elements are mamtamed. 
WRUEJDATA 

^ Communication address of the application seiver sending the write operation 
Communication address of the pritnaty DASD 
Time stamp of receiving the write operation 

AJI irarJrteis of the wite operation including the storage address of where 

the data is to be placed on primaiy storage 
The data to be written by this write operation 

} ■ 



The Data Protect Algorithm 

In diis implementation tins algorithm runs on the protection machine. It operates as 
following: 

^revenfwtte operation received from the application server fio 

^ Aliocate buffer frame within the black box apparatus and return BufferFrame 
a pointer to this buffer 
Write WRITEJDATA to buffer pointed by BufferFrame 

} 

The buffers witiiin tiie black box device are being managed by the data protect 
aigoritfam running in die protection box as a circular buffer as foUowing: 

A Wneate Buffer 

Hfree buffer entries within the buffer exists Tjtjgn 

^ Allocate free entry buffer 

Return BufferFrame pointer to this buffer 
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gise . 

* Locate buffer entry X with data which resides in the buffer for the longest period of time 
Discard the data from buffer entry X 
Allocate free buffer entry X 
Return pointer BufferFrame to buffer frame X 

} 

for data of every write operation stored In the black box apparatus memory Do 

^ Read each WIUTE DATA entry in the order in which It was initially stored 

Based on the storage address, write the data to the appropriate remote DASD location 

> 



3. Claims 

1. A black box apparatus which 

a. Hosts transient write data until it is wntten to remote site 

b. Connects easily to any computing system 

c. Its memory is managed by a computing system it connects too 

d. Host all data protection/data recovery programs 

e. The above programs can be downloaded to any computmg system 

f. Does not need external power supply 

g Can survive any catastrophic disaster , ui i w« 

h. Its internal memory can be removed and installed m another black box 
apparatus without loosing the data it stores 

i. All data and programs stored in it will survive a catastrophic disaster 
j. Equipped with a homing device 

2 A set of algorithms which utilizes the black box apparatus 

a. The algorithms can either run on a separate box or on the same box as 

the mirroring application 

b. Protect data which was not replicated yet to a remote site usmg 
asynchronous mirror scheme m the black box apparatus 

c. Inducing performance impact on the native write operations similar to 
that introduces by asynchronous mirror ^act^ 

d. In a case of a disaster updating off-line the remote mirror DASD with 
all updates which were not yet transmitted to the remote DASD. 

3. A protection machme which connect to the black box and can serve multiple 
application sever by connecting to all of them by a switch 
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